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(54) Method and apparatus for pref iltering of video images 



(57) A method for generating an updated video 
stream from an incoming video stream for use in image 
encoding applications includes the steps of replacing 
each predetermined reference frame amongst a second 
plurality with an updated reference frame, obtained by 
temporal filtering of a first plurality of frames from said 
incoming video stream at positions dependent on the 
position of said each predetermined reference frame. If 

A 



afterwards encoded using an encoding algorithm using 
the updated reference frames as reference frames, the 
thus encoded updated video stream will show a 
reduced peak-to-mean bitrate ratio compared to the 
encoded original video stream. An apparatus for per- 
forming this method is described as well. 
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Description 

The present invention relates to a method for gen- 
erating an updated video stream from an incoming 
video stream for use in image encoding applications s 
and to an apparatus realising such a method, according 
to the preamble of claims 1 and 5. 

Such a method and apparatus are already known in 
the art, e.g. from the European Patent Application 
94 1 14620. 1 " Method for temporal filtering of video $ig- 10 
nals using a motion adaptive spatial filter. In this docu- 
ment a motion-adaptive spatial filtering method and 
apparatus are presented, performing part of front-end 
processing on a video signal which needs to be 
encoded afterwards. The involved filtering steps serve is 
to eliminate or to alleviate temporal noises and perform 
band limitation to thereby improve picture quality and 
encoding efficiency. The temporal filtering has as pur- 
pose to remove noise components in moving areas of 
successive frames, without affecting the details therein. 20 

The filtered signal is then further encoded using 
e.g. standard MPEG-2 encoding techniques. This 
encoding method first identifies so-called reference 
frames in the video stream to be encoded, applies an 
intra-coding algorithm to these reference frames, and 25 
further encodes the other frames differentially with 
respect to these reference frames. Even with the men- 
tioned prior art front-end processing, the resulting bit- 
stream after MPEG-2 encoding remains very bursty, 
resulting in a high peak-to-mean bitrate ratio of the 30 
encoded updated video stream. This high peak-to- 
mean bitrate ratio creates a serious problem for trans- 
mitting these encoded data using ATM transfer capabil- 
ities. This problem could be alleviated by over- 
dimensioning the ATM-network, or by using a flattening 35 
MPEG transport stream, in which the encoded updated 
video elementary stream is packetised into a packet ele- 
mentary stream, and further integrated into a transport 
stream consisting of fixed length packets. Yet also this 
solution results in an inefficient use of the network, 40 
since a lot of these fixed length packets will contain use- 
less bits. 

An object of the present invention is to provide a 
method for generating an updated video stream from an 
incoming video stream for use in image encoding appli- 45 
cations of the above known type, but which is such that, 
after encoding the updated video stream in accordance 
to a particular encoding scheme using reference 
frames, the encoded updated video stream shows 
reduced burstiness, such that it is directly suited for 50 
transmission over an ATM service network. 

According to the invention, this object is achieved 
due to the feet that said method includes the steps as 
stated in claim 1 and that said apparatus realising said 
method is built according to claim 5. ss 

In this way the updated video stream will only con- 
tain updated reference frames, whereas the non-refer- 
ence frames therein have not changed with respect to 



the original video stream. The updated reference frame, 
generated by temporal filtering of a first plurality of 
frames, will deviate from the original reference frame to 
an xtent which depends on the frame evolution over 
time. Indeed, the fester successive frames in the origi- 
nal video stream change over time, the more the 
updated reference frame will be different from the origi- 
nal reference frame. The updated reference frame will 
therefore be discontinuous in the sequence of succes- 
sive frames in the updated video stream, meaning that 
non-reference frames in the vicinity of these updated 
reference frames will deviate much more from the newly 
generated reference frame than they did from the origi- 
nal reference frame. Consequently, further encoding 
non-reference frames with respect to these updated ref- 
erence frames will result in substantially more bits 
required for these non-reference frames, reducing the 
peak-to-mean bitrate ratio of the encoded updated 
video stream. 

An additional characteristic feature of the present 
invention is that said method includes a further spatial 
filtering step as is described in claim 2 and that said 
apparatus further includes spatial filter means as is 
described in claim 6. 

In order to further decrease the already reduced 
peak-to-mean bitrate ratio of the encoded updated 
video signal, an optional spatial filtering step is per- 
formed during generation of the updated reference 
frame. This spatial filtering step further reduces the bit- 
count needed for encoding the reference frame, which 
is also beneficial for reducing the mean bitrate of the 
encoded signal. 

Another characteristic feature of the present inven- 
tion is that said temporal filtering step is performed 
according to claim 3, and that the part of the apparatus 
realising said temporal filtering step is realised accord- 
ing to claim 7. 

This means that the temporal filtering step is made 
dependent upon the content and the content evolution 
of these frames in the original video stream that are 
selected and used for generating the updated reference 
frame. When the selected frames are considerably dif- 
ferent from each other, an adaptive temporal filtering 
step giving more relative importance to these frames 
deviating the most from the original reference frame, will 
in general result in an updated reference frame deviat- 
ing more from the neighbouring frames compared to a 
temporal filtering step in which all selected frames are 
give a fixed weight. Therefore the burstiness of the after- 
wards encoded updated video stream will be reduced 
again. Thus temporal filtering using temporal filter 
parameters that are variable in function of the content 
and the content evolution of the frames selected for per- 
forming this temporal filtering step on, is much more 
effective for reducing the burstiness of the afterwards 
encoded signal, than a temporal filtering step with pre- 
defined constant weighing or filter parameters. 

Another characteristic feature of the present inven- 
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tion is that said spatial filtering step is performed 
according to claim 4, and that the part of the apparatus 
performing said spatial filtering step is realised as stated 
in claim 8. 

In case the frames selected and used for generat- s 
ing the updated reference frame are nearly identical and 
resemble the original reference frame, the previously 
mentioned temporal filtering step will result in nearly the 
same original reference frame, even if an adaptive tem- 
poral filtering step is used. In this case the peak-to- 10 
mean bitrate ratio of the encoded updated signal can 
only be reduced by a spatial filtering step. By then mak- 
ing this spatial filtering step adaptive on the content of 
the selected frames itself, in this sense that images with 
high frequency spatial content will be filtered more than is 
the images with low spatial frequency content, and by 
relatively more filtering spatial content information 
related to the original reference frame, the updated ref- 
erence frame, reduced in bitcount with respect to the 
original one, will also be more deviating from the neigh- 20 
bouring frames than did the original one, thus again 
decreasing the peak-to-mean bitrate ratio of the after- 
wards encoded updated signal. 

The above mentioned and other objects and fea- 
tures of the invention will become more apparent and 25 
the invention itself will be best understood by referring to 
the following description of an embodiment taken in 
conjunction with the accompanying drawings wherein : 

Fig. 1 shows a block scheme of an apparatus 30 
according to the invention, 
Fig. 2 shows part of an embodiment of the appara- 
tus according to the invention. 

Description 35 

The method subject of the invention is a pre- 
processing method for generating an updated video 
stream from an incoming video stream, prior to deliver- 
ing this updated video stream to a video encoder. This 40 
updated video stream being such that, if afterwards 
input to an encoding apparatus realising an encoding 
method making use of reference frames, such as for 
instance the well-known MPEG-2 encoding method, the 
resulting encoded updated video stream being less 45 
bursty than an encoded non-updated video stream. 

The method therefore generates an updated set of 
reference frames, replacing the original reference 
frames. Fig. 1 depicts a block scheme of an apparatus A 
adapted to realise such a method. The incoming video so 
stream is received in the apparatus A via an input termi- 
nal IN. The positions of the reference frames, as chosen 
by the encoding apparatus during the encoding proce- 
dure, are known by a control means CM as second pre- 
determined information, for instance programmabl 55 
during start up. This control means CM forms part of th 
apparatus A and generates a first contr I signal, C1 , 
based on this second predetermined information, and 



on first predetermined information, including informa- 
tion about the positions of the frames of a first plurality, 
to be selected from the original video stream for gener- 
ating one particular updated reference frame. Ing n ral 
this first predetermined information includes the relative 
positions of the frames of the first plurality to be 
selected, with respect to the position of the reference 
frame to be updated. For example, for a particular refer- 
ence frame in the original video stream, the frames of 
the first plurality to be selected can consist of this refer- 
ence frame itself, two reference frames following and 
two reference frames preceding this reference frame. 
The second predetermined information then includes 
the positions of the reference frames, or an algorithm to 
calculate these positions, whereas the first predeter- 
mined information includes the relative positions of the 
frames to be selected per reference frame, in this exam- 
ple being the position of the reference frame itself, the 
positions of the two reference frames preceding and the 
two reference frames following the reference frame 
under consideration, or an algorithm to calculate these 
positions. As is the case for the second predetermined 
information, the first predetermined information can be 
programmable during start up. 

The control means CM delivers this first control sig- 
nal C1 to a selection means SM. This selection means 
SM, also included in the apparatus, is adapted to select, 
for each reference frame from the incoming video 
stream to be updated, a first plurality of frames, the 
position of these selected frames of this first plurality, 
thus being dependent on the position of the reference 
frame to be replaced, as indicated by the first control 
signal C1. 

Such a selection means SM, in some embodiments 
of the apparatus, may therefore include a large memory 
to store large amounts of incoming and/or selected 
frames, depending on the particular selection criterion. 
How to implement such a selection means is however 
commonly known by a person skilled in the art, and will 
therefore not be further described. 

After having selected this first plurality of frames, a 
temporal filtering step is performed on them by a tempo- 
ral filter means TF. Temporal filtering has to be under- 
stood in the most general way as to use the motion 
information contained within the frames included in this 
first plurality, for generating an updated reference frame. 
In this sense, motion-adaptive spatial filtering also has 
to be considered as temporal filtering, as is well known 
by a person skilled in the art Such and other temporal 
filter means have been described rather exhaustively in 
the literature. Possible embodiments consist of a series 
of parallel operating switches, each switch receiving 
one of the selected frames of the first plurality. For such 
embodiments of the temporal filter means, the image 
information pertaining to individual frames of the first 
plurality will flow between th selection means and the 
temporal filter means via a bundle of individual parall I 
links, each link connecting one individual switch of the 
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temporal filter means TP For these embodiments the 
connection b tween the selection means SM and the 
temporal filter means TF of Fig. 1, thus consists of a 
bundle of parallel links. The control of the switches can 
be fixed, or can be made variable upon the image con- 5 
tent of the frames itself. The frames that are passed 
through the switches are then combined in a combining 
unit, also part of the temporal filter means, to generate 
one new image being the updated reference frame. 
However other embodiments are also possible and will 70 
not be further discussed here since these are well 
known to a person skilled in the art. 

The updated reference frame is then forwarded to a 
replacement means RM, also included in the apparatus 
A. This replacement means also receives the incoming is 
original video stream and replaces therein the original 
reference frames with the corresponding updated refer- 
ence frames. To perform this latter function the replace- 
ment means RM is controlled by the control means CM, 
by means of a second control signal C2. This second 20 
control signal C2 informs the replacement means RM 
about the position of the original reference frames in the 
incoming video stream, whereby the replacement 
means can identify these frames and replace them with 
the corresponding updated ones, thereby generating 25 
successive part of the updated video stream. The 
replacement means then passes these successive 
parts of the updated video stream to an output terminal 
OUT. 

Again, depending on the positions of the reference 30 
frames to be replaced, several embodiments of the 
replacement means RM are possible. Since however 
implementing such a means, based on its functional 
description, is rather obvious to a person skilled in the 
art, the details of implementing such a replacement 35 
means will not be described here. It has to be remarked 
that, depending on the processing time needed by the 
temporal filter means, additional buffering memory (not 
shown) may be necessary in the replacement means, or 
in the apparatus in between the input terminal, the 40 
selection means, and the replacement means, to enable 
the apparatus to perform the method properly. 

In this respect, attention must be further drawn to 
the fact that for proper and timely functioning of the 
apparatus, also other control signals, provided by the 45 
control means CM and distributed to all functional 
blocks part of the apparatus, are necessary, although 
they are not shown in Fig. 1 , nor further described here. 
A person skilled in the art is however familiar with these 
principles of operation and knows how to implement so 
them. Since they are not relevant to the invention itself 
they will not be further described. 

The output terminal OUT can be further coupled to 
a video encoding means (not shown) receiving the 
updated video stream for then encoding it, using the 55 
updated reference frames as reference frames during 
the encoding algorithm. 

The method may optionally include a spatial filter- 



ing step, realised by a spatial filter means (not shown on 
Fig. 1). The purpose of this spatial f ilt ring step is to fur- 
ther reduce the bitcount information of the updated 
fram , as generated by the temporal filter means. By 
this, the bitcount information of the updated reference 
frame can be reduced to well below the bitcount infor- 
mation of the original reference frame, thereby further 
reducing the peak-to-mean bitrate ratio of the encoded 
updated video stream, which is beneficial for transport 
over ATM-networks. 

In one variant of the method, realised by one vari- 
ant embodiment of the apparatus (not shown), the spa- 
tial filtering step is performed on the selected frames, 
before temporally filtering them. In this variant embodi- 
ment the spatial filtering means is coupled between the 
selection means and the temporal filter means. In 
another variant of the method the spatial filtering step is 
performed on the result of the temporal filtering step, 
this result being just one frame. This other variant of the 
method is realised by another variant embodiment of 
the apparatus (not shown) wherein the spatial filter 
means is coupled between the temporal filter means 
and the replacing means. In yet another embodiment of 
the apparatus realising a third variant of the method, 
both steps can be intermixed, in this sense that a part of 
the temporal filtering step is performed first on the indi- 
vidually selected frames of the first plurality, resulting in 
a set of intermediate temporally f iltered frames, this step 
being followed by individually spatially filtering these 
intermediate frames, the result of which is then com- 
bined in a combining unit, included in the temporal filter 
means, for generating the updated reference frame. 

This latter variant of the method is realised by an 
embodiment of the apparatus A, part of which is shown 
in Fig. 2. This figure functionally shows how to generate 
an updated reference frame, intended for replacing an 
original reference frame located in the original video 
stream at frame position j. In this embodiment the first 
plurality of frames, selected by the selection means SM 
from the incoming video stream, consists of the frames 
of the original video stream, located at frame positions j- 

bp. Mb-1)p j-2p,j-pj, j+p. j+2p,.... j+fp whereby p 

denotes the period between two successive original ref- 
erence frames in the incoming video stream, and f and 
b are integer numbers indicating the amount of refer- 
ence frames following and preceding reference frame j, 
that are to be taken from this incoming video stream. 
Note that in other embodiments, non-reference frames 
can be selected as well for generating the updated ref- 
erence frame. 

The selection means SM of this embodiment as 
shown in Fig. 2 receives successive frames from the 
original video stream, successively selects therefrom 
successive first pluralities of frames corresponding to 
the particular reference frames, and temporarily stores 
successive first pluralities of frames. For the reference 
fram at position j, the frames from positions j-bp, j-(b- 
1)p j-2p,j-p,j, j+p. j+2p j+fp are thus t mporarily 
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stored in its memory, before passing them in parallel to 
the temporal filter means TF. Also in this embodiment 
the connection between the selection means SM and 
the temporal filter means consists of a bundle of parallel 
links. This temporal filter means TF, in this embodiment 5 
of the apparatus, consists of an amount of switches, 
equal to the amount of frames contained in the selected 
first plurality. In this embodiment of the apparatus the 
control of the switches is variable and depends on the 
image content of the selected frames. This dependency w 
is realised by means of a first control unit, in this embod- 
iment being incorporated in a larger control unit CU. 
Remark that not all temporal filter means require such a 
control unit, which was therefore not added in Fig. 1 
which only depicts a block scheme of the essential ele- 75 
ments of apparatus according to the invention. There- 
fore the control signals generated in this control unit 
which are sent to the temporal filter means are also not 
shown in Fig. 1 . 

The first control unit receives the image content of 20 
the selected frames, and determines, according to a 
first predetermined algorithm or criterion, the control of 
the different switches. Such an algorithm may consist of 
comparing the image content of the selected frames 
with the original reference frame, determining the devia- 25 
tion with respect to the original reference frame, thereby 
giving more relative importance to the most deviating 
frames by closing the corresponding switches and by 
opening the others. Remark that other algorithms are 
also possible. The thus retained frames are subse- 30 
querrtly sent to individual spatial filter means, ail 
together composing a spatial filter means SF of this 
embodiment of the apparatus. All individual spatial fil- 
ters are again controlled by means of separate spatial 
filter parameter, generated by a second control unit in 35 
this particular embodiment again incorporated in the 
larger control unit CU. Based on the spatial image con- 
tent of the selected frames, this control unit determines, 
according to a second algorithm, the individual spatial 
filter parameters which control the individual spatial fil- 40 
ters of the spatial filter means. Such an algorithm may 
consist in determining the high frequency spatial con- 
tent of each of the difference images between the indi- 
vidual selected frames of the first plurality and the 
original reference frame, and adapting the transfer f unc- as 
tion of the individual filters such that the high frequency 
spatial content of the individual frames is separately 
reduced, whereby the most deviating frames are given a 
less rough spatial filtering than the least deviating ones, 
thereby again enhancing the difference between the so 
updated reference frame with the neighbouring non-ref- 
erence frames. This again reduces the burstiness of the 
afterwards encoded updated video signal. 

In this particular embodiment of the apparatus the 
control of both spatial and temporal filter means is even ss 
made dependent on each other, realisable by the fact 
that both spatial and t mporal filter control parameters 
are calculated in one control unit CU, as is depicted in 
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Fig. 2. Such an embodiment allows the updated video 
stream to be optimised for burstiness reduction, while 
maintaining the highest possible quality level at yet rea- 
sonable low mean bit rate. This can be understood from 
two extreme examples from incoming video streams : 
one video stream with nearly identical selected frames, 
and a second incoming video stream with selected 
frames that are seriously different. In the first case there 
is little gain in replacing the original reference frame with 
a weighted average over time of these selected frames. 
An appropriate weak spatial filtering step on the original 
reference frame, adapted by the control unit to not visi- 
bly deteriorate the original video stream quality, while 
still reducing that reference frame's high frequency con- 
tent, is the most obvious solution for generating the 
updated reference frame. This however requires the 
interdependent control of both filters. In the case of seri- 
ously different selected successive frames, the most 
deviating ones should be given a higher weighing factor 
for the newly generated reference frame than the least 
deviation ones. As most energy is concentrated here in 
the time evolution of the original video stream, addi- 
tional strong spatial filtering can be applied in the gener- 
ation of the new reference frame, with little visible 
deterioration of the original video stream quality, this 
however again requiring the interdependency in the 
control of both filters. In both cases, the final result will 
be that non-reference frames in the vicinity differ much 
more from the updated reference frame than they did 
from the original reference frame, thus reducing the 
burstiness after encoding. Besides this the bitcount 
information of the newly generated reference frame will 
not be higher than the bitcount of the original reference 
frame, at yet high quality as is determined by the control 
unit by controlling all filter parameters interdependent^. 

The frames that are passed through the switches 
and underwent a spatial filtering, are then combined in 
a combining unit, CO, included in the temporal filter 
means TF, to generate one updated reference frame at 
position j. This updated reference frame at position j is 
subsequently sent to the replacing means which will 
then further replace the original reference frame at posi- 
tion j by the updated reference frame at position j, and 
perform the same function on all other reference 
frames, as previously described. 

Of course, this embodiment requires the most 
processing power, and is more complex, compared to 
an embodiment with fixed filter control parameters, or 
with spatial and temporal filter parameters that are not 
interdependent. 

If required, the new image sequence can again be 
fed in as input to compute a second iteration new image 
sequence, as part of an iterative procedure. 

From this description it becomes clear that such an 
apparatus A updates the incoming video stream off-line, 
since, depending on the first plurality of frames to b 
selected, rather important memory and processing 
power may be needed. 
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The described apparatus can be implemented in 
hardware or in software, or may employ a combination 
of both to enhance performanc and flexibility. 

While the principles of the invention have been 
described above in connection with specific apparatus, s 
it is to be clearly understood that this description is 
made only by way of example and not as a limitation on 
the scope of the invention. 
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1. Method for generating an updated video stream 
from an incoming video stream for use in image 
encoding applications, said method including a step 

of replacing a frame of said incoming video stream, is 
with an updated frame, obtained by temporal filter- 
ing of a first plurality of frames from said incoming 
video stream at positions dependent on the position 
of said frame, characterised in that said frame is 
obtained by an additional first step of selecting a 20 
second plurality of predetermined reference 
frames, whereby for each predetermined reference 
frame amongst said second plurality, said step of 
updating and replacing is performed, thereby gen- 
erating an updated reference frame related to said 25 
each predetermined reference frame, whereby said 
updated reference frame is to be used during fur- 
ther video encoding of said updated video stream 
as a reference frame for the encoding method, said 
updated video stream being generated as said 30 
incoming video stream with said each predeter- 
mined reference frame replaced by said updated 
reference frame related to said each predetermined 
reference frame. 

35 

2. Method according to claim 1 characterised in that 
said method includes an additional step of spatial 
filtering during generation of said updated refer- 
ence frame. 

40 

3. Method according to claim 1 characterised in that 
said temporal filtering step is performed in accord- 
ance with a plurality of temporal filter parameters, 
whereby the value of each of said temporal filter 
parameters is dependent on said first plurality of 45 
frames. 

4. Method according to claim 2 characterised in that 
said spatial filtering step is performed in accord- 
ance with a plurality of spatial filter parameters, so 
whereby the value of each of said spatial filter 
parameters is dependent on said first plurality of 
frames. 



Apparatus (A) adapted to generate an updated 
video stream from an incoming video stream for 
use in image encoding applications, said apparatus 
(A) including 
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at least one input terminal (IN) adapted to 
receive successive frames of said incoming 
video stream, 

set ction means (SM) coupled to said input ter- 
minal (IN) and adapted to select from said 
incoming video stream, a first plurality of 
frames, upon control of a first control signal 
(C1), 

temporal filter means (TF) coupled to said 
selection means (SM) and adapted to perform 
temporal filtering of said first plurality of frames, 
thereby generating an updated frame, 
replacing means (RM) coupled to said tempo- 
ral filter means (TF) and adapted to replace, 
upon control of a second control signal (C2), a 
frame of said incoming video stream with said 
updated frame, thereby generating a frame of 
said updated video stream, 
an output terminal (OUT), coupled to said 
replacing means and adapted to deliver suc- 
cessive frames of said updated video stream 
control means (CM) adapted to generate said 
first control signal (Ci), and said second con- 
trol signal (C2), based on first predetermined 
information including the relationship between 
the positions of the frames amongst said first 
plurality and the position of said frame to be 
replaced, 

characterised in that 
said control means (CM) is further adapted to 
generate said first control signal (C1) and said 
second control signal (C2) from second prede- 
termined information, including positions of 
predetermined reference frames, included in a 
second plurality of predetermined reference 
frames, 

in that said replacement means (RM) is thereby 
further adapted to replace a predetermined ref- 
erence frame included within said second plu- 
rality, 

in that said temporal filter means (TF) is 
thereby further adapted to generate an 
updated reference frame related to said prede- 
termined reference frame included in said sec- 
ond plurality, 

said updated reference frame thereby gener- 
ated for use as a reference frame during further 
encoding of said updated video stream, 
said updated video stream thereby consisting 
of said incoming video stream whereby said 
each predetermined reference frame is 
replaced by said updated reference frame 
related to said each predetermined reference 
frame. 

Apparatus according to claim 5 characterised in 
that said apparatus further includes spatial filter 
means. 
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7. Apparatus according to claim 5 characterised in 
that said temporal fitter means (TF) is controlled by 
a plurality of temporal filter parameters, generated 
within a first control unit, included in said apparatus, 
said first control unit being adapted to receive said 5 
first plurality of frames of said incoming video 
stream, and to calculate therefrom the value of said 
temporal filter parameters according to a first pre- 
determined algorithm, the value of said temporal fil- 
ter parameters being output signals of said first w 
control unit. 

8. Apparatus according to claim 6 characterised in 
that said spatial filter means is controlled by a plu- 
rality of spatial filter parameters, generated within a is 
second control unit included within said apparatus, 
said second control unit being adapted to receive 
said first plurality of frames of said incoming video 
stream and to calculate therefrom the value of said 
spatial filter parameters according to a second pre- 20 
determined algorithm, the value of said spatial filter 
parameters being output signals of said second 
control unit. 
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